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(54) Image forming apparatus 

(57) An image forming process for easily making an 
image drawing (ortho-image) in an actual place while 
confirming it in real time. Measuring of control points (at 
least three points) is performed (S1 10), and preparation 
is made for forming an ortho-image based on image 
data thereof and survey data. Where photographing is 
carried out from a plurality of directions, additional 
image measuring is performed (S140). Then, ortho- 
image formation is performed on a PC (S160). Here, if a 
formed ortho-image is not what is desired, ortho-image 
correction is performed (S180). After a satisfactory 
image is obtained (S200), the process moves to a next 
area to be measured, and the same operation is 
repeated. On the other hand, unless a satisfactory 
image is obtained, additional image measuring (S140) 
is performed again. 
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Description 

BACKGROUND OF THE INVENTION 

s [0001] The present invention relates to an image forming apparatus, an image forming method and a computer-read- 
able storage medium having an image forming program. More particularly, the present invention relates to a portable 
and simple image measuring technology utilized for surveying and making a drawing in a land surveying field, which 
uses digital processing by digital photogrammetry and a personal computer. 

[0002] Specifically, the present invention relates to an image forming technology for forming a digital orthogonally pro- 
10 jected image (ortho-image) from one to a plurality of images to be superposed on the drawing when a drawing of a site 
to be measured is made. The present invention relates to an image forming technology which enables any operator to 
easily make a drawing of an orthogonally projected image by using a survey instrument and images so as to stick an 
ortho-image to the drawing, and also enables a digital orthogonally projected image to be generated and rectified such 
that the situation of a site to be measured can be understood in detail. An object of the present invention is to provide 
75 an image which enables stable stereo analysis to be performed even for a stereo-image. 

[0003] The present invention can also be applied to, for example, an image obtained by photographing a wide range 
divided into small images having overlapping areas with one another. 

[0004] According to a prior art, a drawing obtained by surveying in a site to be measured has been made by using 
paper and a pencil or the like which are typically used in plane table surveying. In recent years, a line drawing of a site 
20 to be measured has been made by using a survey instrument and a portable computer in combination which are typi- 
cally used in pen-based mapping system. 

[0005] Fig. 1 8 is a view illustrating conventional stereo- image measuring. Usually, as shown in the drawing, the three- 
dimensional coordinates are obtained by photographing two or more overlapped images based on the principle of tri- 
angulation by stereo-image photographing. If a target range is wide, a plurality of images are photographed. For exam- 
25 pie, in even a simple site, photographing of ten to twenty images is usually necessary. In this case, unless orientation 
(i.e., calculating a camera position, inclination or the like) can be performed for each stereo pair (model), a stereo-image 
cannot be rectified. Three-dimensional measuring cannot be performed, either. 

SUMMARY OF THE INVENTION 

30 

[0006] In the conventional case, however, a drawing was expressed only by a line image even if it was made in the 
actual place, and it was impossible to sufficiently understand the situation of the site. Thus, the situation of the site was 
photographed by a camera. However, since photographing was performed from an oblique direction whereas a direc- 
tion of a requested drawing was vertical, a photographed image had no direct connection with the drawing. As a result, 
35 it was necessary for an operator to understand the situation of the site by comparing the drawing with the image. This 
practice was inconvenient, and understanding of the site was difficult. 

[0007] A range to be photographed with one image was limited. Even in the case of photographing a plurality of 
images, because correlation and continuity among the respective photographs are lacking, this led to the difficulty of 
obtaining consistency among the photographs, and comparison with the drawing was complicated and difficult. If a 

40 highly accurate ortho-image was required, lens distortion of the camera was an obstacle. Conventionally, it was impos- 
sible to produce a highly accurate ortho-image by a camera which has no data on lens distortion. 
[0008] In the case of conventional land surveying, for surveying a site to be photographed and making a drawing for 
the site, it was necessary to survey an enormous number of points and perform three-dimensional measuring. Thus, 
much man power and labor had to be expended. On the other hand, if a dose and terrestrial photogrammetry is used, 

45 indirect survey three-dimensional data can be obtained only by photographing. But since only a camera using a film was 
available conventionally developing and scanning took time (e.g., two to three days), and several days were also 
expended for analyzing work. Unless processing up to analysis was carried out after film development, there was no 
knowing whether photographed image data was analyzable or not, and whether stable, sure and highly reliable analysis 
was possible or not. As a result, conventionally, this method has not been used so often because the necessity to again 

so perform the work such as photographing may happen. 

[0009] With popularized and increased use of digital cameras in recent years, digital close and terrestrial photogram- 
metry can now be performed by a digital camera which is different from a conventional film or analog camera. Use of 
this technology eliminates labor and time for film development and scanning. Since analysis is performed by a compu- 
ter, the process from photographing to analyzing can be finished within one to two days. 

55 [0010] However, after acquisition of photographic data, it was necessary to bring the photographic data to the place 
of an analyzing device (computer) for analysis and measuring. Thus, photographing improper for analysis might result 
in the necessity to again perform the work such as photographing, and stable and sure analysis was not always carried 
out. 
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[001 1 ] For performing analysis and measuring by images, if control point/orienting point arrangement, a photographic 
range, photographic overlapping or the like was not proper, stable and highly reliable analysis was impossible. Conse- 
quently, unstable results occurred, and even analysis itself was impossible. 

[0012] The present invention was made with the foregoing points in mind. The object of the present invention is to 
s provide an image forming apparatus for, when a drawing of the situation of a site to be measured is made, easily making 
a drawing of the site from an orthogonally projected image (ortho-image) so as to facilitate understanding of the situa- 
tion. The object of the present invention is to quickly and easily generate/rectify an image in the site to be measured 
without any storage omission or mistakes, to understand the situation of the site on the spot, and to easily make an 
image drawing (ortho-image) in real time while confirming the same. The object of the present invention is to perform 
10 image complementary measuring only by simple photographing and surveying of several points with a survey instru- 
ment, to simultaneously obtain an image drawing for facilitating understanding of the situation of the site, to interpolate 
portions not photographed or hard to make out, and to produce an ortho-image of a high resolution and wide range. 
[0013] Another object of the present invention is to facilitate understanding of a situation from a plurality of images, 
even if the situation is hard to grasp, for example, some parts are difficult to see or a far part becomes rough with one 
75 image, and produce a highly accurate ortho-image. The object of the present invention is to integrate a plurality of 
images by repeating simple work and obtain a wide-ranging ortho-image. Further, the object of the present invention is 
to form a highly accurate and high-quality ortho-image at a high speed by photographing a plurality of images over- 
lapped with adjacent areas. 

[0014] The still another object of the present invention is to correct lens distortion (camera calibration correction) by 
20 simple measuring, and simultaneously form a highly accurate ortho-image even by a camera having no lens distortion 
data. Further, the object of the present invention is to form a good quality ortho-image of a necessary place with 
required accuracy by using various survey instruments to rectify the ortho-image while measuring the three-dimen- 
sional coordinates. 

[001 5] According to the present invention, the following operations and confirmation can be performed in a site to be 
25 photographed: 

(1) confirmation of a measuring range (e.g.. overlapping); 

(2) generation and confirmation of an ortho-image obtained by integrating plurality of images; 

(3) confirmation of the entire measuring/photographing range and a range to be drawn. 
so (4) confirmation of entire arrangement of control points/orienting points; and 

(5) confirmation of orientation, and confirmation of a model formed for stereo method analysis. 

[001 6] In accordance with first solving means of the present invention, there is provided an image forming apparatus, 
which comprises: 

35 

a control point measuring section for measuring a centrally projected image having a plurality of control points, and 
obtaining the image coordinates for said control points; 

a coordinate transformation parameter calculating section for obtaining a transformation parameter for correlating, 
based on said image coordinates for said control points obtained by said control point measuring section and the 
40 three-dimensional coordinates for actually measured control points, said image coordinates with said three-dimen- 
sional coordinates; 

an orthogonally projected image (ortho-image) forming section for forming an orthogonally projected image from 
said centrally projected image based on said transformation parameter obtained by said coordinate transformation 
parameter calculating section; and 
45 an ortho-image correcting section for correcting the image coordinates obtained by said ortho-image forming sec- 

tion based on the three-dimensional coordinates for actually measured additional points, and then performing cor- 
rection of said orthogonally projected image. 

[0017] In accordance with second solving means of the present invention, there are provided an image forming 
so method, an image forming apparatus and a computer-readable storage medium having an image forming program. In 
this case, the image forming method comprises: 

an image input function, to which a plurality of images including control points or orienting points overlapped with 
one another are inputted; 

55 a storage function for previously storing ground coordinate values of control points or orienting points; 

an orthogonally projected image (ortho-image) forming function for forming an ortho-image from plurality of images 
inputted by said image input function based on image coordinate values, alternatively photographic coordinate 
value, and said ground coordinate values of the control points or the orienting points; and 
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a determining function for determining, based on the ortho-image formed by said ortho-image forming function, 
necessity of re-photographing, and necessity of changing a photographing position, alternatively a control point or 
an orienting point position. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0018] 

Fig. 1 is a constitutional view of an image forming apparatus of the present invention. 

Fig. 2 is a flow chart of an image forming process according to a first embodiment of the present invention. 

Fig. 3 is an explanatory view showing a case where a wide range is measured. 

Fig. 4 is a flow chart of an image forming process according to a second embodiment o1 the present invention. 
Fig. 5 is a flow chart of on-line control point measuring. 

Fig. 6 is a flow chart of automatic control point measuring performed by a control point measuring section. 

Fig. 7 is an explanatory view showing an example of control point arrangement. 

Fig. 8 is a flow chart of coordinate transformation parameter calculation process. 

Fig. 9 is a flow chart of additional image measuring. 

Fig. 10 is a flow chart of an ortho-image forming process. 

Fig. 1 1 is a flow chart of ortho-image correction process. 

Fig. 12 is a flow chart of displaying a control point shortage area or an improper image area. 

Fig. 13 is an explanatory view showing an example of a control point shortage area or an improper image area. 

Fig. 14 is a flow chart of displaying an image shortage place or an improper place. 

Fig. 15 is an explanatory view showing displaying of an image shortage place or an improper place. 

Fig. 16 is an explanatory view showing the camera coordinates and the model coordinates. 

Fig. 17 is a flow chart of off-line processing. 

Fig. 18 is an explanatory view showing conventional image measuring and photographing. 
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

[0019] Now, the preferred embodiments of the present invention will be described with datum to the accompanying 
drawings. 

A. Constitution and operation of image forming apparatus 

[0020] Fig. 1 is a constitutional view of an image forming apparatus of the present invention. The image forming appa- 
ratus comprises a control unit 10, a storage section 2, an input/output interface 3, a three-dimensional coordinates input 
section 7, an image input section 4, a display section 5, and an input/output section 6. The control unit 10 furthermore 
includes a control point measuring section 101, a coordinate transformation parameter calculating section 102, an 
ortho-image forming section 103, an ortho-image correcting section 104, and an additional image measuring section 
105. These sections (functions) can be realized by, for example, a portable computer (a PC). 

[0021] The storage section 2 previously stores control point data, and stores image data or the like. The control point 
data is stored in, for example, the three-dimensional coordinates (X, Y, Z) called ground coordinates. The input/output 
interface 3 is composed of a common bus or the like, and used for connecting various devices. The three-dimensional 
coordinates input section 7 obtains the three-dimensional coordinates for control points or additional measuring points 
by various survey instruments, e.g., a global positioning system (a GPS), a total station, and so on, or a three-dimen- 
sional coordinates measuring device. The image input section 4 obtains a two<limensional or three-dimensional image 
from, for example, a digital camera. From the image input section 4, one or a plurality of images including control points 
are inputted. 

[0022] The display section 5 is provided with a CRT, a liquid crystal display a plasma display or the like, and performs 
two-dimensional or three-dimensional displaying. The display section 5 displays an ortho-image formed by the ortho- 
image forming section 1 03, the ortho-image correcting section 1 04 of the control unit 1 0. The display section 5 also dis- 
plays a control point position inputted by the control point measuring section 101, an image added by the additional 
image measuring section 105 or the like. The input/output section 6 transfers various image data and results of deter- 
mination, alternatively the three-dimensional coordinates data or the like from a three-dimensional coordinates input 
device such as a survey instrument or the like with other units. For the input/output section 6, various input/output 
devices, for example, an optical disk device, a card recording medium (e.g., an HDD, a memory or the like), a floppy 
disk, a keyboard, a pen, a mouse, a terminal and a CD-ROM disk drive, can be provided. The input/output section 6 
gives instructions regarding control points and various positions on the screen of the display section 5 by a pointing 
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device such as a mouse, a write pen or the like. 

[0023] The control point measuring section 101 measures a centrally projected image having a plurality of imprinted 
control points, and obtains the image coordinates for the control points. The control point measuring section 101 meas- 
ures, by using, as the three-dimensional coordinate input section 7, a survey instrument for measuring a distance and 
5 an angle from an already-known point, the three-dimensional coordinates for the control points based on the obtained 
coordinates, and stores the same in the storage section 2 as required. Depending on the control points, the control point 
measuring section 1 01 can automatically obtain each centrally projected image coordinate value corresponding to each 
control point for which the three-dimensional coordinates has been obtained. 

[0024] The coordinate transformation parameter calculating section 102 obtains a transformation parameter. This 

10 transformation parameter is used for transforming the image coordinates obtained by the control point measuring sec- 
tion 101 into the photographic coordinates having an original point set as a principal point (a point where a line passing 
through a lens center and orthogonal to the screen intersects the image), and correlating, based on the image coordi- 
nates and the three-dimensional coordinates for the control points, which have been obtained by the control point 
measuring section 101, the image coordinates with the three-dimensional coordinates. Also, as described later, the 

is coordinate transformation parameter calculating section 102 can calculate, by satisfying a collinear condition for central 
projection and using an image input section where a principaJ distance is known, a transformation parameter based on 
at least three control points, for which a ground coordinate system is known. Further, the coordinate transformation 
parameter calculating section 102 can correct lens distortion based on lens distortion data of the image input section 
or actually measured data of a plurality of control points. 

20 [0025] The ortho-image forming section (ortho-image formng function) 1 03 transforms the three-dimensional coordi- 
nates into the image coordinates based on the transformation parameter obtained by the coordinate transformation 
parameter calculating section 102, and forms an orthogonally projected image. For forming an orthogonally projected 
image by sticking a plurality of images, the ortho-image forming section 103 can select each image obtained based on 
a distance from each image to a control point or a distance from each image to a measuring point. 

25 [0026] Also, the ortho-image forming section 103 forms an ortho-image from image coordinate values (or photo- 
graphic coordinate values) of the control points/orienting points and ground coordinate values based on a plurality of 
stereo images inputted by the image input section 4. The ortho-image forming section 103 can include a function of dis- 
criminately displaying a non-overlapped area/non-photographed area determined by the ortho-image correcting section 
104 or a shortage/failed positioning of the control points/orienting points on the display screen of the ortho-image. Fur- 

30 ther, the ortho-image forming section 103 can include a function of executing orientation such as successive orienta- 
tion, relative orientation or the like based on a plurality of stereo images. 
. [0027] The ortho-image correcting section 104 corrects the image coordinates obtained by the ortho-image forming 
section 103 based on the three-dimensional coordinates for actually measured additional points, and performs correc- 
tion of the orthogonally projected image. Also, the ortho-image correcting section 104 can form, if a plurality of centrally 

35 projected images are obtained, an orthogonally projected image by combining images selected according to a proper 
standard, e.g., images of places having specified/lower reduction scales or having relatively small reduction scales, 
alternatively images close to the measuring point or the control point, on priority basis. Further, the ortho-image cor- 
recting section 104 can display on the display section 5, if a plurality of centrally projected images are obtained, an 
image shortage area or an improper area which results from formation of the orthogonally projected image performed 

40 by the ortho-image forming section 103. 

[0028] A determining function included in the ortho-image correcting section 104 determines, based on the ortho- 
image formed by the ortho-image forming section 103, necessity of changing a photographing position and necessity 
of changing positions of the control points/orienting points. This determining function includes a function of extracting a 
non-overlapped area, the area being not overlapped by at least two stereo images in the ortho-image. A determining 

45 section 12 includes a function of extracting a non -photographed area which is not covered with by stereo images in a 
measuring target area. Also, the determining function includes a function of determining a shortage/failed positioning 
of control points/orienting points included in a portion overlapped by at least two stereo images in the ortho-image. Fur- 
ther, the determining function includes a function of outputting, if results of orientation performed by the ortho-image 
forming section 1 1 are not converged, an instruction for changing setting of the control points/orienting points to both or 

so either of the ortho-image forming section 1 03 and the additional image measuring section 105. 

[0029] A correction data forming function included in the ortho-image correcting section 1 04 forms data for re-photo- 
graphing if the determining function detects a non-overlapped area/non-photographed area or a shortage/failed posi- 
tioning of the control points/orienting points in the ortho-image. This data for re-photographing is necessary for solving 
the detected failure, and for this data, for example, data regarding a photographing position, data regarding a photo- 

55 graphing range, data regarding selection of control points/orienting points, data for specifying control points/orienting 
points to be rearranged or the like can be used. Such data is fed back to the ortho-image forming section 103. 
[0030] The additional image measuring section 105 measures other centrally projected images so as to include the 
control points measured by the control point measuring section 101, and calculates transformation parameters for these 
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centrally projected images by using the coordinate transformation parameter calculating section 102. 
[0031] Next, an overall operation will be summarized. 

[0032] Fig. 2 is a flow chart of an image forming process according to a first embodiment of the present invention. The 
drawing shows a case where an image is formed based on one or more images and three-dimensional coordinate val- 
5 ues by a survey instrument. An image forming process carried out in the control unit 10 is described hereinbelow by 
referring to Fig. 2. 

[0033] The process shown in Fig. 2 has two courses of processing: on-line processing performed in an actual place, 
and off-line processing for performing only control point measuring in the actual place and bringing back remaining jobs 
to be finished in an office or the like. In the on-line processing, different from the case of the off-line processing, control 

10 point measuring, subsequent image forming, displaying, and so on, are carried out in the actual place. 

[0034] In the image forming process, first measuring of control points (three points or more) is performed (step S1 1 0) , 
and preparation is made for forming an ortho-image based on image data thereof and survey data. Furthermore, if pho- 
tographing is carried out from a plurality of directions, additional image measuring is performed (step S140). For per- 
forming analysis only with one image, the additional image measuring (step S140) is unnecessary Then, for example, 

15 on a PC, ortho-image formation is performed (step S160). Here, if a formed ortho-image is not what is desired, ortho- 
image correction is performed (step S180). In this ortho-image correction work, determination is made as to whether a 
satisfactory image is obtained or not (step S200). lf"OK", then the process moves to a next area to be measured, and 
the same operation is repeated from the control point measuring (step S110). On the other hand, if no satisfactory 
image is obtained in the ortho-image correction work (step S180). then the additional image measuring (step S140) is 

20 carried out again, and the foregoing operations are repeated until a satisfactory image is obtained. 

[0035] The additional image measuring (step S140) is performed, for example if some parts are hard to see when 
photographing is carried out from one direction, if an image of a far part becomes rough even on one image and thus 
sufficient accuracy is not provided, or the like. In other words, if an ortho-image is formed by performing further photo- 
graphing from another direction for compensating for an already inputted image or yet another direction where there is 

25 a shortage, a place hard to see or a place of less accuracy will be selected/synthesized from a camera image of another 
direction, and thereby a good quality ortho-image will be formed. 

[0036] Fig. 3 is a view illustrating a case where a wide range is measured. If a range to be measured is wide, a meas- 
uring range may be decided in accordance with necessary accuracy, and the image forming process in Fig. 2 may be 
repeated sequentially in respective areas of measuring ranges 1, 2, 3, • • • and so on, of Fig. 3. Necessary accuracy 
30 depends on, for example, a distance from an object, a size of one CCD pixel or the like. Thus, by dividing a wide range 
for processing, an ortho-image can be; easily formed no matter how wide a range is. 

[0037] Fig. 4 is a flow chart of an image forming process according to a second embodiment of the present invention. 
The process shown is image measuring processing for performing stable measuring by stereo images. Image measur- 
ing processing is described hereinbelow by referring to Fig. 4. 

35 [0038] First, control points/orienting points are set, and measured data is set by a survey instrument (step S1 100). 
The control points have been accurately measured on the basis of a ground coordinate system (absolute coordinate 
system) (X, Y, Z) beforehand by the survey instrument. If the orienting points are arranged uniformly (discretely) on a 
photographic image, it is not necessary to measure positions thereof beforehand. Needless to say, the orienting points 
can be also used as position-measured control points. The ground coordinate system (X, Y, Z) is a three-dimensional 

40 orthogonal coordinate system where real space of a model has been set. 

[0039] Then, photographing is performed such that each image can sufficiently overlap another, and the control 
points/orienting points, for example at least six points or more, can overlap one another (step S1200). Here, as an 
example, each image has at least six or more control points/orienting points, however, the proper number thereof can 
be used as occasion demands. For example, if bundle adjustment or the like is used, the number of control points for 

45 measuring can be reduced to three. Even if orienting points are used instead of control points, the necessary number 
of targets is about six. Alternatively, four points are enough if orienting points can be assumed that they are arranged 
on a plane. 

[0040] After the photographing, various set values necessary for image processing, e.g., both or either of an ortho- 
image forming range and resolution, and so on, are set onto the portable computer (a PC) (step S1300). Then, the con- 
so trol points/orienting points of each image are measured on a PC (step S1400). Here, by the ortho-image forming func- 
tion, orientation and ortho-image formation are automatically carried out according to a program (step S1500). The 
specific processing will be described in detail later. 

[0041 ] After the calculation by the ortho-image forming function, the results of orientation and an ortho-image are dis- 
played on the display screen (step S1600). Then, based on the results of step S1600. each check item is confirmed for 
55 the measured image (step S1 700). 

[0042] The display function displays arrangement of all and each image by the ortho-image. Furthermore, the insuf- 
ficient results of orientation are discriminate! y displayed based on various bits of display information by which the check 
items are confirmed. Regarding the check items, if the results of orientation are insufficient (e.g.. targetted accuracy is 
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not achieved, no convergence occurs or the like), improper selection or arrangement of the control points/orienting 
points is determined. A failure of an entire measuring range such as existence of a non-photographed area caused by 
omission of photographing or the like is confirmed by the formed ortho-image. If a non-overlapped area exists because 
of an image overlapping failure, propriety of an overlapped portion is confirmed. If there is such an overlapping failure, 

5 by extracting only an overlapped part between adjacent images in the formed ortho-image and displaying the same, a 
gap is formed in a joined part between images on the ortho-image, and thus immediate determination can be made. 
[0043] In addition to the confirmation of the display screen by a measuring operator, the check items can be automat- 
ically confirmed by the control function (especially, the determining function). If the results of orientation do not con- 
verge as in the case of failed arrangement or failed positioning of the control points, or if successive orientation is not 

10 normally carried out, the results of orientation containing errors are outputted in the step S1600 is outputted, and a fail- 
ure such as an improper image is discriminately displayed. For example, the control function can allow the display func- 
tion to automatically display the fact that the results of orientation fail to achieve the targetted accuracy or fail to 
converge by an identifier, a flag, and can also allow the display function to discriminately display a proper range by color 
or patterns. Even in the case where a non -photographed area or a non-overlapped area exists, the area can be properly 

is recognized by the control function and using an image processing technology. Specifically, for example, a vacant place 
of an image is detected, and by identifying an area/position thereof, the above failures can be determined. 
[0044] If necessity of correction is decided based on the results of checking, then a decision is made as to necessity 
of determining, because of a shortage, a positioning failure of the control points/orienting points, arrangement thereof 
again for re-photographing (step Si 720). 

20 [0045] Here, for example, if rearrangement of the control points/orienting points is not necessary because only an 
image photographing position is not proper, as in the case where a non -photographed area or a non-overlapped area 
exists, data for re-photographing is formed by the correction data forming function, and photographing is performed 
again according to this data (step S1860). Then, the process is resumed again from the step S1400. 
[0046] On the other hand, for example, if the arrangement of the control points/orienting points must be changed as 

25 in the case where the results of orientation are not converged or successive orientation is not carried out normally, rear- 
rangement thereof is examined (step S1 740). If there are other control points/orienting points on the image and the 
image coordinates for these points is obtained without re-photographing, selection of the control points/orienting points 
is changed from the already photographed image (step S1760), and the above calculation is started from the step 
St 500. This processing may be performed automatically. On the other hand, if other proper control points/orienting 

30 points do not exist on the image, arrangement changing is performed by moving the arrangement of the control 
points/orienting points (step S1840). Then, data for re-photographing the image including the rearranged control 
points/orienting points is formed by the correction data forming function, and re-photographing is performed according 
to the formed data (step S1860). Then, the process is executed from the step S1400. 

[0047] The re-photographing processing in the step S1860, the measuring processing m the step S1400 may be exe- 
35 cuted only for the image including the control points/orienting points, which is necessary to handle each failure, alter- 
natively for all the images to be photographed. 

[0048] The decision as to necessity of re-photographing based on the results of checking, and the decision as to the 
step to which the process returns when re-photographing is necessary, may be manually made by the operator based 
on the display screen, alternatively made automatically by a PC, otherwise made by proper combination of the manual 
40 operation and a PC. 

[0049] As described above, when it Is decided that any correction is necessary correction is determined based on the 
result of checking in the step S1700, the process is finished with the decision that a desired ortho-image has been 
obtained. 

45 B. Detail of the processing 

[0050] Next, the flow chart of Fig. 2 will be described in detail. 

(1) Control point measuring processing by the control point measuring section 101 (step S110). 

50 

[0051] Hereinbelow, control point measuring processing (step S1 10) by the control point measuring section 101 is 
described in detail. 

(1-1) On-line measuring 

55 

[0052] First, an example of on-line measuring will be described. Fig. 5 is a flow chart of on-line control point measur- 
ing. In on-line control point measuring for forming an ortho-image on the field of a work site, work is done while forming 
and confirming an ortho-image in an actual place. Thus, according to the on-line control point measuring, non-efficient 
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work such as re-measuring the control points in the actual place can be avoided, such work being necessary in the case 
of measuring mistakes or shortage parts found after a photographed image is brought back and analyzed. 
[0053] First, control points, three points at the minimum, are set in a range to be measured (step S1 12). Furthermore, 
the control points are preferably be discriminable targets on an image and measured by a survey instrument (three- 
5 dimensional coordinate input section 7). For the control points, for example, prisms or reflecting targets can be used, 
alternatively those tartgets which return for returning reflecting (measuring) signals can be used in the case of a non- 
prism total station. Then, photographing is performed by a digital camera (image input section 4) (step S1 14). This pho- 
tographing may be performed for one image, or the desired number of images from a plurality of directions. For exam- 
ple, when some parts are hard to see if photographed from one direction, when image accuracy is considered, it is 
w better to perform further photographing from a plurality of directions. 

[0054] Then, the image photographed by the digital camera and stored in the storage device of the camera is trans- 
ferred to the control unit 10 of the image forming apparatus and inputted to the storage section 2 (step S116). The 
image inputted to the control unit 1 0 comes to be measured on a PC, and the photographed image is displayed on the 
display section 5. Here, the targets are collimated by the survey instrument (step S1 1 8) and confined. If OK, the control 
15 points as targets are measured on the image on a PC (step S120). In this case, for example, positions of the control 
points are instructed by a pen as the input device of the input/output section 6. After the image coordinates (px, py) 
(CCD coordinate system, and an original point is set to be a screen end such as the left upper end of the screen) is 
measured, a measuring instruction is outputted to the three-dimensional coordinates input section 7, measuring is per- 
formed by the three-dimensional coordinates input section 7 (step S1 22), and then measured ground coordinate values 
20 of the control points are sent through the input/output l/F 3 to the control unit 10 and the storage section 2 (step S124). 
Accordingly, correlation of the ground coordinates for the control points with the image coordinates on a PC and meas- 
uring are completed. Then, the same work is repeated for the minimum three points (step S126). 
[0055] Next, automatic control point measuring by the control point measuring section 101 will be described. 
[0056] Fig. 6 is a flow chart of automatic control point measuring performed by the control point measuring section. 
25 Fig. 7 is a view showing an example of arrangement of control points. 

[0057] First, control points having reflecting sheets are set (step S501). Attaching the reflecting sheets facilitates auto- 
matic measuring with an image and simultaneous measuring work by a total station, a non-prism total station. The con- 
trol points are set as targets to be seen from any directions, and these points can be fixed to, for example, apexes of 
triangle pyramids or triangle poles. If the number of control points is three, for example, arrangement like that shown in 
30 the drawing is set, and it is advisable to set one obviously different in size. Then, as one example, photographing is car- 
ried out with a strobe light (step S503). Accordingly, any situations can be surely detected. Then, the photographed 
image is inputted to the personal computer (step S505). Then, the three-dimensional coordinates are measured by the 
survey instrument (step S507). For measuring the control point having the largest reflecting sheet, information thereof 
is transferred as an attribute is simultaneously transferred from the survey instrument to the control unit 10. 
35 [0058] Then, by template matching (image correlation processing), the image coordinates on the image is detected 
(step S509). For example, a light quantity returns from each of the reflecting sheets has high luminance, and by regis- 
tering the shapes of the control points as templates (matching patterns) beforehand, position detection is facilitated. For 
the templates, in this example, two kinds of small and large templates can be prepared. Then, the image coordinates 
for the position-detected control points are correlated with points of the three-dimensional coordinates (step S51 1). For 
40 example, by arranging three points and setting one among them to be large as shown in the drawing, the large control 
point on the image becomes discriminable by the template, and is correlated based on the attribute measured on the 
three-dimensional coordinates. The other two points can be correlated based on the three-dimensional coordinates 
positions and the image coordinates positions thereof even if these points are photographed from any directions. 
[0059] For increasing reliability of position detection, two images are photographed, one with a strobe light and the 
45 other without a strobe light, and by taking a difference between these two images, only the targets emerge. Thus, auto- 
matic coordinate position detection by matching becomes surer and more reliable. Accordingly, detection can be surely 
carried out from any directions. 

[0060] The processing has been described with datum to the large and small reflecting sheets. Image coordinates 
detection and correlating can also be performed by coloring or patterning the reflecting sheets. Other than the reflecting 
so sheets, any discriminal materials on the image can be used. Accordingly, there are no limitations placed on the number, 
arrangement, shapes, and so on of control point. 
[0061] Next, template matching processing will be described. 

[0062] For template matching, any proper method such as a normalized cross correlation method, a sequential sim- 
ilarity detection algorithm (SSDA) or the like may be used. For example, by using the SSDA, a processing speed can 
55 be increased. Hereinbelow. the SSDA is described. 

[0063] An expression of the SSDA is shown below. In this expression, a point where a residual R(a,b) becomes small- 
est is an image position to be obtained. For increasing a processing speed, for example, in addition of this expression, 
the addition is finished when a value of R(a,b) exceeds a minimum value of a past residual, and calculation is performed 
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so as to move to next (a.b). 



N1-1 W1-1 

R(a,b)= £ S I '(a,b) ( m v"i) - T(m^ n A ) | (1) 

T(m 1p n-j): Template image, 
l (a b) (m 1 , n-,): Partial image of target image 
(a,b): Left upper coordinates of template image 
10 R(a,b): Residual 

(1-2) Coordinate transformation parameter calculation processing by the coordinate transformation parameter calculat- 
ing section 102 (step S300) 

15 [0064] If the foregoing measuring for the control points is OK, then coordinate transformation parameter calculation 
by the coordinate transformation parameter calculating section 102 is performed (step S300). 
[0065] Fig. 8 is a flow chart of coordinate transformation parameter calculation process. 

[0066] First, since the points measured on a PC are obtained as the image coordinates on a charge coupled device 
(a CCD) such as a digital camera, these coordinates are transformed into a photographic coordinate system (x, y) (step 
20 S203). The photographic coordinate system (x, y) is the two-dimensional coordinates having an original point set as a 
principal point. On the other hand, the image coordinates are CCD coordinate system (px, py), for example, the two- 
dimensional coordinates having an original point set in the left upper side. 

[0067] The three points measured on the ground coordinate system (X, Y, Z) by measuring of the survey instrument 
and the two points of the image coordinate system (x, y) measured on a PC are substituted for expressions below, and 
25 each parameter for coordinate transformation (coordinate transformation parameter) is obtained (step S205). Expres- 
sions 2 and 3 are collinear condition expressions where a projection center, an image on a CCD and an object are 
located on a straight line. Accordingly, if there are at least three already-known points, each coordinate transformation 
parameter can be calculated. In this case, however, a principal distance c must be nearly known. If a calibrated camera 
is used, no problem occurs since a principal distance c is known. 

30 

a „ (X-X 0 )+a 12 ( V- y 0 ) + a 13 (Z-Z Q ) 
X "° a 31 (X-X 0 ) + a 32 (/-/ 0 )+a 33 (Z-Z 0 ) { ] 

35 _ a 21 (X-X 0 )+a 22 ( Y- Y Q )+a 23 (Z-Z 0 ) 

y -' c a 3 ,(X-X Q )+a 32 (Y-Y 0 )+a 3Z (Z-Z 0 ) { } 



Herein, 




x, y 


: I mage coordinates, 


xo.yo 


: Principal point position, 


X, Y,2 


: Ground coordinates, 




: Coordinates of projection center, 


c 


: Principal distance 


ay 


: Rotation matrix 


CO, d>, K 


: Inclination of camera 



[0068] On the other hand, if a principal distance c is not known, this value can be obtained by measuring coordinate 
so values of four points on a plane based on a two-dimensional projective transformation. Also, by using six points, a rela- 
tionship between the image coordinates and the three-dimensional coordinates (target points coordinates) of the object 
can be obtained by an approximate solution based on a three-dimensional projective transformation expression (direct 
linear transformation). 

[0069] Hereinbelow, a method by the three-dimensional transformation expression is described. This processing is 
55 for calculating a transformation parameter for obtaining a pixel position on an ortho-image. Here, the processing is per- 
formed mainly based on known coordinates for the control points/orienting points and the measured image (photo- 
graphic) coordinates. 

[0070] First, a basic expression is shown below. 
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X - L 9 X+L 10 V+L^ Z+1 

^5^+^-6 /+/ -7 Z+L 8 
y= LgX+L^V+L^Z+1 



Herein, 

TO 

(x,y) : Image coordinates, 

(X.Y.Z) :Ground coordinates, 

L-| to L-j-j :Coordinates transformation parameters 



15 [0071] By calculating the expression 4 by using a least squares method based on data of the control points, coordi- 
nate transformation parameters to L-n for deciding a relationship between the image coordinates (x, y) and the 
ground coordinates (X,Y,Z) can be obtained. 

[0072] Further, a coordinate transformation expression is not limited to the foregoing, and any proper coordinate 
transformation expressions can be employed as long as these can correlate the ground coordinates with the image 
20 coordinates. 



(1-3) Camera calibration 



[0073] The foregoing description has been made of the case where measuring is performed with accuracy for permit- 
25 ting lens distortion to be ignored or a non-distortion lens is used. However, if there is lens distortion and this distortion 
cannot be ignored for accuracy measuring is carried out with a camera having pre-calculated lens distortion or while 
correcting lens distortion by processing described below. 

[0074] Specifically, in the case of using a camera having no lens distortion data, the lens distortion is corrected by 
measuring six or more control points and obtaining lens distortion of the camera based on the following expressions 5 
30 to 7. In the case of a camera having pre-calculated lens distortion, coordinate values are obtained while correcting the 
lens distortion by using these expressions from the start. 



a u (X-X 0 )+a, 2 (Y-Y 0 )+a, 3 (Z-Z 0 ) 
X "° a 3 ,{X-X 0 ) + a 32 (Y-Y 0 )+a 33 {Z-Z 0 ) + X 



(5) 



a 21 (X-X 0 )+a 22 (y-y 0 )+a 23 (Z-Z 0 ) 
y '° a 3 ,(X-X 0 )+a 32 (Y-Y 0 )+a 33 {Z-Z 0 ) +Ay 



(6) 



40 



45 



Ax = x 0 + x(/c 1 r 2 + k 2 r 4 ) 



r 2 = (x 2 + y 2 )/c 2 



(7) 



50 



55 



k n , k 2 : parameters of radical distortion 

[0075] These expressions are calculated by measuring six or more control points on the ground coordinates and the 
image coordinates and by a successive approximation solution. 

[0076] In addition to the foregoing, by using a direct linear transformation with self -calibration, correction of lens dis- 
tortion can be performed by expressions 8 and 9. In this case, however, at least eight control points are necessary. 

L,X+L 2 Y+L 3 Z+L 4 
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y + Ay= L S X+L ' Y+L ? Z+L * (9) 

s [0077] By canceling denominators of the above expressions based on data of the control points and by using the least 
squares method, unknown variables k 1( k 2 , x 0 and y 0 can be obtained. 

[0078] As described above, lens distortion is obtained by the foregoing expressions, and accordingly, measuring while 
correcting lens distortion can be performed. These expressions are only examples, and calculation may be carried out 
by other expressions. 

10 [0079] Then, referring again to Fig. 5, after the foregoing coordinate transformation parameters calculation processing 
(step S300) is finished, by using the transformation parameters, a residual between the coordinate transformation val- 
ues and the control points is obtained, and a decision is made as to whether the residual is within a prescribed value or 
not (step S128). If the residual is within the prescribed value, then the process moves to the next processing. On the 
other hand, if the residual is not within the prescribed value, the process returns to the step S1 18, and while increasing 

is the number of points to be measured, the operations of the steps S11 8 to S128 are performed until the residual comes 
within the prescribed value. 

[0080] Here, the residual can be obtained in the following manner. Specifically, the transformation parameters 
obtained by the expressions 2 and 3 (alternatively the expressions 5 and 6) and the image coordinates (x, y) measured 
for the control points are substituted in expressions 10 and 1 1 described below, and calculation control points X' and Y' 
20 of the ground coordinate system (X, Y, Z) are calculated. Then, a residual with the actually measured ground coordinate 
values (X, Y) are obtained by an expression 12. If the residual 8 thus obtained is within the prescribed value, it is OK. 
For the prescribed value, for example, accuracy necessary in an actual place is set. Herein, n means the number of con- 
trol points. Expressions for the residual should not be limited to the ones described here, but any other expressions can 
be used. 

25 

X' = (Z-Z 0 ) 11 21 ; 31 + X 0 (10) 

a 13 x+a 23>^ a 33 C 

30 r = (z-z 0 ) a ™ x+az2y ' a3zC + Y 0 (11) 



35 



55 



6 = Jz(X'-X) 2 +Z( Y'-Y) 2 I n (12) 
(2) Additional image measuring processing by the additional image measuring section 105 (step S140) 



[0081] Next, additional image measuring processing (step S140) will be described in detail. Fig. 9 is a flow chart of 
additional measuring. If the number of images to be photographed is one, alternatively if an image is formed with one 

40 image temporarily, this processing can be skipped. Also, if a plurality of images are photographed first in steps S1 14 
and S1 16, a step of photographing (step S142) and a step of inputting an image to a PC (step S144) can be skipped. 
[0082] Photographing of additional images is performed so as to include all the control points set by the control point 
setting (step S1 12) carried out in the control point measuring processing (step S1 10). Then, the images are inputted to 
a PC (step S1 44), the additionally photographed images are displayed on the display section 5, and an additional image 

45 to be measured is selected (step S146). Then, the control points imprinted on the selected image are measured on a 
PC (step S148). This work is repeated by a number of times equal to the number of control points (step S150). For 
example, if a portable computer of a pen input type is used, by indicating the control points as targets by a pointing 
device such as a pen, the image coordinates (px, py) (or the photographic coordinates (x, y)) thereof is obtained. Then, 
coordinate transformation parameter calculation like that described above is performed (step S300). 

so [0083] Further, for additionally measuring other images, the process returns to the additional image section (step 
S146), and this processing is repeated by a number of times equal to the number of images to be photographed. When 
no additional measuring is carried out, the process moves to next step to form an ortho-image (step S160). 



(3) Ortho-image formation processing by the ortho-image forming section 103 (step S160) 

[0084] Next, ortho-image formation processing (step S160) will be described in detail. Fig. 10 is a flow chart of an 
image forming processing. 

[0085] First, a ground coordinate for each pixel on an ortho-image is calculated (step S207). In this processing, for 
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forming an ortho-image, the image coordinates (x, y) for the ortho-image are transformed into the ground coordinates 
(X, Y, Z). The ground coordinates (X, Y, Z) is calculated by using the transformation parameters previously obtained in 
the step S205 of the coordinate transformation parameter calculation processing (step S300). In other words, the 
ground coordinates (X, Y, Z) corresponding to the image coordinates (x, y) for the ortho-image is provided by the follow- 
5 ing expression. Accordingly, a position for obtaining each pixel on the ortho-image can be obtained. 

X=X 0 + xAX 
Y=Y 0 - y t,Y 

10 

Zm ,aX + bY + d (13) 



15 Herein, 

(X 0 , Y 0 ) :Position of left upper side of ortho-image in ground coordinate system, 
(AX, AY): Size of one pixel in ground coordinate system (e.g., m/pixel), 
(x, y) : Image coordinates for ortho-image 
20 (X, Y, Z) :Ground coordinates, 

a, b, c, d : Coefficients of plane equation formed by a plurality of control points interpolating certain image coordi- 
nates (x, y). 

[0086] This time, by using the transformation parameters obtained in step S205, the image coordinates (x, y) corre- 
25 sponding to the ground coordinates (X, Y, Z) obtained in step S207 is calculated based on the expressions 2 and 3 or 
the expressions 5 and 6 (step S209). From the obtained image coordinates (x, y), a gray value on the ground coordi- 
nates (X, Y, Z) for the proper image is obtained. This gray value is a pixel gray level in a two-dimensional position (X, Y) 
on the orthoHmage. In this way, a gray value of an image to be stuck to a position (X, Y) on the ground coordinates is 
obtained. By performing the foregoing processing for all the pixels of the ortho-image, image sticking is carried out (step 
30 S211). 

[0087] Hereinbelow, sticking of a plurality of images is described more. 

[0088] If the number of images is not singular but plural, then selection of an image is decided based on a positional 
relationship between the control points and the camera. In other words, a position (X 0 , Y 0> Z 0 ) of a camera for each 
image is calculated based on the expressions 2 and 3. Accordingly, a distance is obtained from values thereof and actu- 
35 ally measured values (X, Y, Z) of control point coordinates, a pixel gray value of a closer image is obtained and then the 
image is stuck. Thus, by carrying out image sticking by using an image closer to the control point, image data having 
high resolution can be selected. 

(4) Ortho-image correction processing by the ortho-image correcting section 104 (step S180) 

40 

[0089] Next, ortho-image correction (step S1 80) will be described in detail. Fig. 1 1 is a flow chart of ortho-image cor- 
rection. 

[0090] Ortho-image formation (step S160) is performed, checking is made on the formed ortho-image visually or by 
later described processing. If there are measuring mistaken parts or parts which especially need measuring data, 
45 measuring is performed for these parts the survey instrument (step S1 84). The three-dimensional coordinates (ground 
coordinates) measured is transferred from the survey instrument to the control unit 10 (step S188), the above-described 
ortho-image formation (step S160) is automatically executed, immediate displaying of the ortho-image is carried out by 
the display section 5, and then checking is made again. 

[0091] Accordingly, since measuring is performed point by point and processing is performed while confirming a cor- 
so rected image in real time, measuring mistakes or measuring omissions can be prevented, and thus measuring can be 
always performed while confirming an image as an end product. 

[0092] Next, a flow illustrating a method of extracting and correcting a control point shortage place or an improper 
place from the formed ortho-image will be described. 

[0093] Fig. 12 shows a flow chart of control point shortage place or improper image place displaying. Fig. 13 is an 
55 explanatory view showing an example of a control point shortage area or an improper image area. 

[0094] First, control point shortage place displaying is executed. The ortho-image formed from one or a plurality of 
images is first displayed (step S601). Then, a range in a measuring area which is intended to be confirmed is specified 
(step S603). Here, whether control points are located in a proper control points arranging area or not is checked, and 
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displayed (step S605). For example, if the number of control points is six, a display is like that shown in Fig. 13. For 
checking control point arrangement, a proper control point range is divided on the ortho-image based on the number of 
control points to be measured, the coordinates of the control points are checked, decision is made as to whether the 
control points are located in the divided sections or not, and a result of the decision is displayed. Alternatively, a frame 
5 of a proper control point range like that shown in Fig. 13 (in this example, the area is divided into six sections) is dis- 
played. But no limitation is placed on a method for deciding a proper control point range. 

[0095] Subsequently, displaying of an improper image place is executed. First, a gap and a difference of elevation 
between adjacent control points are calculated (step S607). Here, a point of which a calculated value (e.g., a difference 
of elevation) is equal to or higher than a threshold value is displayed so as to show the fact (step S609). The threshold 

10 value is calculated from an image reduction scale such that a drastically changed point can be detected from, for exam- 
1 pie, the number of control points or a gap and a difference of elevation between the control points. 

[0096] Then, as described above, in ortho-image correction (step S180), a control point shortage part and a part in 
the vicinity of the point of which the calculation value is equal to or higher than the threshold value are measured by the 
survey instrument, and then the image is corrected. If no satisfactory result is obtained even by these procedures, it is 

15 advisable to execute additional image measuring based on the information regarding these displayed images. 

[0097] By the above measuring and processing, the control point shortage part or the improper part of the ortho- 
image can be corrected. 

[0098] Next, a method for extracting the shortage part or the improper part from the formed ortho-image and display- 
ing the same will be described. Hereinbelow, a method of specifying a shortage/improper part based on image resolu- 
20 tion will be described. - 

[0099] Fig. 1 4 shows a flow chart of displaying an image shortage part/improper part. Fig. 1 5 is an explanatory view 
showing displaying of an image shortage part/improper part. Necessary measuring accuracy is set according to an 
object as it differs from object to object. 

[0100] First, the ortho-image formed from one or a plurality of images is displayed (step S701). Here, cases of per- 
25 forming photographing from positions A and B are assumed. Then, a range in a measuring area which is intended to 
be confirmed is specified (step S703). Since a camera position (Xq, Y 0 , Zq) has been obtained based on the expres- 
sions 2 and 3, accuracy of one pixel in the measuring confirming area is calculated from this position (step S705). If 
photographing is performed from plural directions, the above calculation is executed for each camera. Then, a place 
where pixel accuracy of each camera does not achieve set accuracy and a range for overlapping are displayed on the 
30 ortho-image (step S707). Then, additional image measuring is performed from the direction of the displayed place (step 
S709). Herein, a case of photographing additional images from an additional image photographing position C is 
described. 

[0101] Accordingly, an image for compensating for the image resolution (one pixel accuracy) of the image shortage 
place/improper place is obtained and, as a result, a satisfactory image can be obtained. The pixel accuracy can be 
35 determined, for example, in a manner that accuracy is reduced as a ratio of transforming a centrally projected image 
into an ortho-image is increased, or as a distance from an image photographing point to an object having an imprinted 
image is increased. 

(5) Orientation 

40 

[0102] Next, orientation of each model will be described in detail. The orientation is, for example, a part of processing 
in step S1500 shown in Fig. 4. 

[0103] Based on the three-dimensional coordinates of each image obtained in step S1400 shown in Fig. 4, orientation 
can be sequentially performed from the measuring points (e.g., six or more measuring points) such as the control 
45 points/orienting points imprinted in each model image. The orientation is data used for checking in step S1700. 

[0104] First, as an example, relative orientation is described. For stereo images, positions of left and right cameras 
are respectively obtained by calculation shown below. 

[0105] Fig. 16 is an explanatory view showing the camera coordinates and the model coordinates. 
[0106] In the camera coordinate system, a lens center (projection center, principal center) is set as an original point, 
so and x and y axes are in parallel with x and y axes of the photographic coordinates. If a principal distance is c (c>0), then 
a point on a photographic plane is expressed by (x, y, - c). The model coordinate system (X, Y, Z) is a three-dimensional 
coordinate system for defining a stereo image formed from a pair of two stereo photographs, and an original point 
thereof is set in the projection center of the left photograph. 

[0107] First, parameters are calculated under a coplanarity condition expression like the following. 
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(14) 



15 



20 



25 



Herein, 

X 01. Y 01- Z 01 
X 02. Y 02» Z 02 

X2, Yg, Z2 



Projection center coordinates of left image expressed by model coordinate system, 
Projection center coordinates of right image expressed by model coordinate system, 
Image coordinates of left image expressed by model coordinate system, 
Image coordinates of right image expressed by model coordinate system. 



[0108] Now, as shown in the drawing, an original point of the model coordinate system is set in the projection center 
of the left side, and a line connecting the projection center of the right side is set in the X axis. For a scale, a base line 
length is set to a unit length. Parameters obtained at this time are five rotational angles, i.e., a rotational angle of the 
Z axis and a rotational angle 4>i of the Y axis of the left camera, and a rotational angle k 2 of the Z axis, a rotational angle 
<j> 2 of the Y axis and a rotational angle <*2 of the X axis of the right camera. In this case, it is not necessary to consider 
a rotational angle co-, of the X axis of the left camera as it is zero. 

[0109] Under the above condition, a coplanarity condition expression 15 becomes one like that shown below, and 
each parameter can be obtained by calculating this expression. 



30 



F(* v ♦ 1- K 2' ♦ 2' ® 2) = 



Y1 Z, 
Y 2 Z 2 



=YiZ 2 -Y 2 Z 1= 0 



(15) 



35 



[0110] Herein, between the model coordinates (X, Y, Z) and the camera coordinates (x, y, z), a relational expression 
of coordinate transformation like that shown below is established. 
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[0111] By using the foregoing expressions, unknown parameters are obtained based on the following procedure. 

55 

1. An initial approximate value is usually set to 0. 

2. A value of differential coefficient is obtained by the expression 7 when the coplanarity condition expression 
(expression 13) is tailor-developed around the approximate value and made linearization, and then an observation 
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equation is established. 

3. By using the least squares method, a correction amount is obtained for the approximate value. 

4. The approximate value is corrected. 

5. By using the corrected approximate value, operations from 2 to 5 are repeated until results converge. 

5 

[0112] No convergence may occur, for example, as in the case where arrangement of the control points, orienting 
points is improper. In such a case, in the displaying of orientation results in step S1600 of Fig. 4, an improper image is 
discriminated, and then displayed. 

[01 1 3] If the results of calculating unknown parameters converge, another step, for example, successive orienting, is 
io performed. The successive orientation is for setting an identical coordinate system by unifying inclinations or reduction 
scales of the respective models subjected to orientation and interconnecting a plurality of images. In the successive 
orientation, a pair of stereophotographs are connected to each other by fixing one orientation parameter and operating 
only the other orientation parameter. If this processing is carried out, a connection range represented by the following 
expression is calculated. 

15 

AX j = (X jr -X jl )/(Z o-Z j( ) 
^ j =(Y jr -Y j( )/(Z 0 -Z jl ) 
20 AZj-(Z jr-Z j,)/(Z o-Z ji) 

AD j = J(AXj 2 + A Y j 2 ) (17) 

Herein, 

25 

(AXj-, AjjYAZj-i): Hh left model of course coordinate system 
(AXj r AY ir AZj r ): j-th right model of course coordinate system 

[0114] If AZj and ADj are equal to a predetermined value (e.g., 0.0005 (1/2000)) or lower as a result of calculation, it 
30 is decided that the successive orientation is normally executed. 

[0115] If results of calculation do not converge, or if normal execution of the successive orientation is not decided, 
then an error is outputted in the orientation result display in the step S1600 of Fig. 4, and a failure, such as an improper 
image or the like, is discriminably displayed. 

35 C.Application 

(1) Use of various survey instruments 

[0116] Next, use of various survey instruments and advantages will be described. 
AO [0117] For example, if correction work is carried out by the non-prism total station, in the case of an object which 
returns a light (measuring signal), an ortho-images is renewed as occasion demands only by performing measuring for 
the object. Accordingly, use of such a survey instruments is extremely effective for facilitating work and providing capa- 
bility of measuring a number of points. 

[0118] Use of another survey instruments is very effective in that if correction processing is carried out by an auto- 
45 tracking total station, only by walking with a prism in a shortage measuring area, an ortho-image can be formed with 
higher accuracy. 

[011 9] Use of the other survey instruments is also greatly effective in that if an area can receive date from satellites, 
only by walking with a GPS in the area, an ortho-image is renewed, and any operator can form an ortho-image while 
easily performing complementary measuring. 

so [0120] For obtaining a high-density and highly accurate ortho-image without any failures, it is advisable to perform 
stereo photographing to overlap ranges to be measured, and perform relative orientation an actual place. In this way, 
automatic stereo matching (image correlation processing) is performed from stereo images to obtain high-density 
three-dimensional measuring points and, by forming an image by the ortho-image forming section, high-density and 
highly accurate ortho-image can be obtained. Alternatively, a shortage part, a discontinuous part or the like is manually 

55 measured on the display while seeing formed stereo images, and by combining the measuring points thereof and then 
forming an ortho-image by the ortho-image forming section, an ortho-image having high luminance can be obtained 
without any failure. 
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(2) Measuring of wide-ranging area 

[0121] A wider range can be measured by executing, after the measuring for the measuring range 1 shown in Fig. 3 
is over, the process again from START of the image formation shown in Fig. 2. In this case, no special consideration is 

5 necessary for a boundary between the ranges to be measured. Alternatively, by intentionally overlapping adjacent 
ranges, e.g., the measuring ranges 1 and 2 with each other, and placing some control points in the area thereof control 
point measuring of these points by the survey instrument can be omitted, and it is only necessary to measure images 
of these points (step S148). Also, by increasing an overlapping rate, images can be stereo-measured by using the sec- 
ond embodiment of the present invention. Accordingly by using the images and control points of the measuring ranges, 

10 1 , 2, • • • so as to form an ortho-image in each area, an ortho-image unifying the measuring ranges 1 , 2, • • • can be 
easily formed. Thus, gradual expansion of area to be measured can be facilitated. 

(3) Off-line measuring 

15 [0122] The control point measuring (step S110) was an example for performing on-line measuring of control points. 
Next, off-line processing will be described. Fig. 17 shows a flow chart of off-line processing. 

[0123] In the case of off-line processing, only photographing and control point measuring by the survey instrument 
are performed in an actual place, and all other operations are done off-line (e.g., in an office or the like). Off-line meas- 
uring in control point measuring (step S1 10) is divided into, for example, image photographing and control point meas- 
20 uring, and these operations are carried out separately in a job site. Results of these operations are then used for 
performing thorough analysis in the office or the like. In this case, a plurality of images are obtained beforehand, and 
image formation is performed later while selecting a proper image. 

[0124] Accordingly, for the control point measuring, the survey instrument and a PC are not associated with each 
other (in other words, a PC is unnecessary), and measuring is carried out only with the survey instrument (step S132). 
25 Then, control point data obtained by the measuring are batch-transferred to a PC (step S1 36). Then, additional image 
measuring (step S140) is performed. Thus, in the case of off-line processing, since control point measuring and photo- 
graphing are carried out separately, an image obtained by aerial photographing performed by a helicopter, a balloon or 
the like can be processed. 

[0125] An image measuring program for executing the foregoing image measuring method of the present invention 
30 can be provided by a recording medium such as a CD-ROM, a floppy disk or the like. 

Claims 

1 . An image forming apparatus comprising 

35 

a control point measuring section for measuring a centrally projected image having a plurality of control points, 
and obtaining the image coordinates for said control points; 

a coordinate transformation parameter calculating section for obtaining a transformation parameter for corre- 
lating, based on said image coordinates for said control points obtained by said control point measuring section 
40 and the three-dimensional coordinates for actually measured control points, said image coordinates with said 

three-dimensional coordinates; 

an orthogonally projected image (ortho-image) forming section for forming an orthogonally projected image 
from said centrally projected image based on said transformation parameter obtained by said coordinate trans- 
formation parameter calculating section; and 
45 an ortho-image correcting section for correcting the image coordinates obtained by said ortho-image forming 

section based on the three-dimensional coordinates for actually measured additional points, and then perform- 
ing correction of said orthogonally projected image. 

2. An image forming apparatus according to claim 1 , further comprising 

an additional image measuring section for measuring another centrally projected image to include said control 
points measured by said control point measuring section, and calculating a coordinate transformation param- 
eter for the measured another centrally projected image by using said coordinate transformation parameter 
calculating section. 

55 

3. An image forming apparatus according claim 1 or 2, wherein 

said measured control points are measured by a survey instrument as a three-dimensional coordinate input 
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section for measuring a distance and an angle from an already-known point alternatively by a global position- 
ing system. 

4. An image forming apparatus according to one of the claims 1 -3, wherein 

said control point measuring section automatically obtains each image coordinate value of a centrally projected 
image corresponding to each control point for which the three-dimensional coordinates are obtained. 

5. An image forming apparatus according to one of the claims 1 -4, wherein 

said coordinate transformation parameter calculating section calculates, by satisfying a collinear condition for 
central projection and using an image input section in which a principal distance is known, a transformation 
parameter based on at least three control points for which ground coordinates are known. 

6. An image forming apparatus according to one of the claims 1 -5, wherein 

said coordinate transformation parameter calculating section corrects lens distortion based on lens distortion 
data of the image input section, alternatively actually measured data regarding the plurality of control points. 

7. An image forming apparatus according to one of the claims 1-6, wherein 

said ortho-image forming section selects, for forming an orthogonally projected image by sticking a plurality of 
images, each obtained image based on a distance from each image to a control point alternatively a distance 
from each image to a measuring point. 

8. An image forming apparatus according to one of the claims 1 -6. wherein 

said ortho-image correcting section forms, where a plurality of centrally projected images are obtained, an 
orthogonally projected image by combining images of places having reduction scales equal to or lower than a 
predetermined reduction scales or having relatively small reduction scales, alternatively images close to a 
measuring position or a control point, on priority basis. 

9. An image forming apparatus according to one of the claims 1 -6, wherein 

said ortho-image correcting section displays, where a plurality of centrally projected images are obtained, an 
shortage place or an improper place of image, which results from formation of the orthogonally projected 
image by said ortho-image forming section. 

10. An image forming apparatus according to one of the claims 1 -9, wherein 

for measuring a wide-ranging area, by using parts of control points located in first and second measuring 
ranges in common, an orthogonally projected image is formed in each measuring range. 

11. An image forming method comprising: 

an image input function, to which a plurality of images including control points or orienting points overlapped 
with one another are inputted; 

a storage function for previously storing ground coordinate values of control points or orienting points; 

an orthogonally projected image (ortho-image) forming function for forming an ortho-image from plurality of 

images inputted by said image input function based on image coordinate values, alternatively photographic 

coordinate value, and said ground coordinate values of the control points or the orienting points; and 

a determining function for determining, based on the ortho-image formed by said ortho-image forming function, 

necessity of re-photographing, and necessity of changing a photographing position, alternatively a control 

point or an orienting point position. 

12. An image forming method according to claim 1 1 , further comprising: 

a display function for displaying the ortho-image formed by said ortho-image forming function. 
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13. An image forming method according to claim 11 or 12, wherein 

said image input function includes a function of inputting stereo images as a plurality of images, and 
said determining function includes a function of extracting a non-overlapped area, the area being not over- 
5 lapped by the stereo images in said ortho-image. 

14. An image forming method according to one of the claims 11-13, wherein 

said determining function includes a function of determining a shortage or a failed positioning, of control points 
70 or orienting points included in portions overlapped by stereo images in said ortho-image. 

15. An image forming method according to one of the claims 11-13, wherein 

said determining function includes a function of extracting a non -photographed area in a measuring target 
15 area, which is not covered with images in said ortho-image. 

16. An image forming method according to one of the claims 11-15, wherein 

said ortho-image forming function includes a function of discriminatably displaying a non-overlapped area or a 
20 non-photographed area determined by said determining function, alternatively a shortage or a failed position- 

ing of the control points or the orienting points on a display screen of said ortho-image. 

17. An image forming method according to one of the claims 11-16, wherein 

25 said determining function further includes a correction data forming function of forming data for re-photograph- 

ing, where a non-overlapped area or a non-photographed area, alternatively a shortage or a failed positioning 
of the control points or the orienting points, is detected in said ortho-image. 

18. An image forming method according to one of the claims 11-17, wherein 

30 

said ortho-image forming function includes a function of executing orientation such as successive orientation, 
relative orientation or the like based on a plurality of images, and 

said determining function includes a function of outputting, if results of said orientation do not converge, an 
instruction to change setting the control points or the orienting points to said ortho-image forming function. 

35 

19. An image forming apparatus comprising: 

means for achieving an image forming method according to claim 11. 

40 20. A computer-readable storage medium having an image forming program for achieving an image forming method 
according to claim 1 1 . 
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